10 Gigabit Ethernet

The 10 gigabit Ethernet (10GE or 10GbE or 10 GigE) computer networking standard was first published in 2002. It defines a version of Ethernet with a nominal data rate of 10 Gbit/s (billion bits per second), ten times faster than gigabit Ethernet.

10 gigabit Ethernet defines only full duplex point to point links which are generally connected by network switches. Half duplex operation, hubs and CSMA/CD (carrier sense multiple access with collision detection) do not exist in 10GbE.

The 10 gigabit Ethernet standard encompasses a number of different physical layer (PHY) standards. A networking device may support different PHY types through pluggable PHY modules, such as those based on SFP+. Over time market forces will determine the most popular 10GE PHY types.[1]

At the time that the 10 gigabit Ethernet standard was developed, interest in 10GbE as a wide area network (WAN) transport led to the introduction of a WAN PHY for 10GbE. This operates at a slightly slower data-rate than the local area network (LAN) PHY and adds some extra encapsulation. Both share the same physical medium-dependent sublayers so can use the same optics.

In 2007 one million 10GbE ports were shipped, in 2009 two million ports were shipped and in 2010 over three million ports were shipped.[2][3]

Contents

Standards

Over the years the Institute of Electrical and Electronics Engineers (IEEE) 802.3 working group has published several standards relating to 10GbE. These included: 802.3ae-2002 (fiber -SR, -LR, -ER and -LX4 PMDs), 802.3ak-2004 (-CX4 copper twin-ax InfiniBand type cable), 802.3an-2006 (10GBASE-T copper twisted pair), 802.3ap-2007 (copper backplane -KR and -KX4 PMDs) and 802.3aq-2006 (fiber -LRM PMD with enhanced equalization).

The 802.3ae-2002 and 802.3ak-2004 amendments were consolidated into the IEEE 802.3-2005 standard. IEEE 802.3-2005 and the other amendments were consolidated into IEEE Std 802.3-2008.

Physical layer modules

To support different 10GbE physical layer standards, many interfaces consist of a standard socket into which different PHY modules may be plugged. Physical layer modules are not specified in an official standards body but by multi-source agreements (MSAs) that can be negotiated more quickly. Relevant MSAs for 10GbE include XENPAK (and related X2 and XPAK), XFP and SFP+. When choosing a PHY module, a designer considers cost, reach, media type, power consumption, and size (form factor).

XENPAK was the first MSA for 10GE and had the largest form factor. X2 and XPAK were later competing standards with smaller form factors. X2 and XPAK have not been as successful in the market as XENPAK. XFP came after X2 and XPAK and it is also smaller.

The newest module standard is the enhanced small form-factor pluggable transceiver, generally called SFP+. Based on the small form-factor pluggable transceiver (SFP) and developed by the ANSI T11 fibre channel group, it is smaller still and lower power than XFP. SFP+ has become the most popular socket on 10GE systems.[4][5] SFP+ modules do only optical to electrical conversion, no clock and data recovery, putting a higher burden on the host's channel equalization. SFP+ modules share a common physical form factor with legacy SFP modules, allowing higher port density than XFP and the re-use of existing designs for 24 or 48 ports in a 19" rack width blade.

Optical modules are connected to a host by either a XAUI, XFI or SFI interface. XENPAK, X2, and XPAK modules use XAUI to connect to their hosts. XAUI (XGXS) uses a four-lane data channel and is specified in IEEE 802.3 Clause 48. XFP modules use a XFI interface and SFP+ modules use an SFI interface. XFI and SFI use a single lane data channel and the encoding specified in IEEE 802.3 Clause 49.

SFP+ modules can further be grouped into two types of host interfaces: linear or limiting. Limiting modules are preferred except when using old fiber infrastructure which requires the use of the linear interface provided by 10GBASE-LRM modules.[6]

Optical fiber

There are two classifications for optical fiber: single-mode (SMF) and multi-mode (MMF).[7] In SMF light follows a single path through the fiber while in MMF it takes multiple paths resulting in differential mode delay (DMD). SMF is used for long distance communication and MMF is used for distances of less than 300 m. SMF has a narrower core (8.3 µm) which requires a more precise termination and connection method. MMF has a wider core (50 or 62.5 µm). The advantage of MMF is that it can be driven by low cost VCSEL lasers for short distances, and multimode connectors are cheaper and easier to terminate reliably in the field. The advantage of SMF is that it can work over longer distances.[8]

In the 802.3 standard reference is made to FDDI-grade MMF fiber. This has a 62.5 µm core and a minimum modal bandwidth of 160 MHz*km at 850 nm. It was originally installed in the early 1990s for FDDI and 100BaseFX networks. The 802.3 standard also references ISO/IEC 11801 which specifies OM1, OM2, OM3 and OM4 MMF fiber types. OM1 has a 62.5 µm core while the others have a 50 µm core. At 850 nm the minimum modal bandwidth of OM1 is 200 MHz*km, of OM2 500 MHz*km, of OM3 2000 MHz*km and of OM4 4700 MHz*km. FDDI-grade cable is now obsolete and new structured cabling installations use either OM3 or OM4 cabling. OM3 cable can carry 10GbE 300 metres using low cost 10GBASE-SR optics (OM4 can manage 400 metres) .[9][10]

To distinguish SMF from MMF cables, SMF cables are usually yellow, while MMF cables are orange (OM1 & OM2) or aqua (OM3 & OM4) however it must be noted that in fibre optics there is no agreed colour for any specific optical speed or technology with the exception being angular physical connector (APC), it being an agreed colour of green.[11]

There are also active optical cables. These have the optical electronics already connected eliminating the connectors between the cable and the optical module. They plug into standard optical module sockets. They are lower cost than other optical solutions because the manufacturer can match the electronics to the required length and type of cable.

10GBASE-SR

10GBASE-SR ("short range") is a port type for multi-mode fiber and uses 850 nm lasers. Its Physical Coding Sublayer 64B/66B PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 Gbit/s.

Over obsolete FDDI-grade 62.5 micron multi-mode fiber cabling it has a maximum range of 26 metres, over 62.5 micron OM1 it has a range of 33 metres, over 50 micron OM2 a range of 82 metres, over OM3 300 metres and over OM4 400 metres.[12] [10] OM3 and OM4 are the preferred choices for structured optical cabling within buildings. MMF has the advantage over SMF of having lower cost connectors because of its wider core.

The 10GBASE-SR transmitter is implemented with a vertical-cavity surface-emitting laser (VCSEL) which is low cost and low power. OM3 and OM4 optical cabling is sometimes described as laser optimized because they have been designed to work with VCSELs. 10GBASE-SR delivers the lowest cost, lowest power and smallest form factor optical modules.

For 2011, 10GBASE-SR is projected to make up a quarter of the total 10GbE adapter ports shipped.[13]

There is a non-standard lower cost, lower power variant sometimes referred to as 10GBASE-SRL (10GBASE-SR lite). This is inter-operable with 10GBASE-SR but only has a reach of 100 metres.

10GBASE-LR

10GBASE-LR ("long reach") is a port type for single-mode fiber and uses 1310 nm lasers. Its Physical Coding Sublayer 64B/66B PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 Gbit/s.

10GBASE-LR has a specified reach of 10 kilometres (6.2 mi), but 10GBASE-LR optical modules can often manage distances of up to 25 kilometres (16 mi) with no data loss.

The 10GBASE-LR transmitter is implemented with a Fabry–Pérot or Distributed feedback laser (DFB). DFB lasers are more expensive than VCSELs but their high power and longer wavelength allow efficient coupling into the small core of single mode fiber over greater distances.

10GBASE-LRM

10GBASE-LRM, (Long Reach Multimode) originally specified in IEEE 802.3aq is a port type for multi-mode fiber and uses 1310 nm lasers. Its Physical Coding Sublayer 64B/66B PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 68. It delivers serialized data at a line rate of 10.3125 Gbit/s.

10GBASE-LRM supports distances up to 220 metres (720 ft) on FDDI-grade multi-mode fiber and the same 220m maximum reach on OM1, OM2 and OM3 fiber types[10]. 10GBASE-LRM reach is not quite as far as the older 10GBASE-LX4 standard.

Some 10GBASE-LRM transceivers also support distances up to 300 metres (980 ft) on standard single-mode fiber (SMF, G.652), however this is not part of the IEEE or MSA specification.

10GBASE-LRM uses electronic dispersion control (EDC) for receive equalization[14].

10GBASE-LRM has been a failure in the market.[13]

10GBASE-ER

10GBASE-ER ("extended reach") is a port type for single-mode fiber and uses 1550 nm lasers. Its Physical Coding Sublayer 64B/66B PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 Gbit/s.

The 10GBASE-ER transmitter is implemented with an externally modulated laser (EML).

10GBASE-ER has a reach of 40 kilometres (25 mi) over engineered links and 30 km over standard links.[15][10]

10GBASE-ZR

Several manufacturers have introduced 80 km (50 mi) range ER pluggable interfaces under the name 10GBASE-ZR. This 80 km PHY is not specified within the IEEE 802.3ae standard and manufacturers have created their own specifications based upon the 80 km PHY described in the OC-192/STM-64 SDH/SONET specifications.

The 802.3 standard will not be amended to cover the ZR PHY.

10GBASE-LX4

10GBASE-LX4 is a port type for multi-mode fiber and single-mode fiber. It uses four separate laser sources operating at 3.125 Gbit/s and coarse WDM with four unique wavelengths around 1310 nm. Its Physical Coding Sublayer 8B10B PCS is defined in IEEE 802.3 Clause 48 and its Physical Medium Dependent PMD in Clause 53.[10]

It supports a range of 300 metres (980 ft) over FDDI-grade, OM1, OM2 and OM3 multi-mode cabling (all these fiber types are specified to have a minimum modal bandwidth of 500 MHz*km at 1300 nm).

10GBASE-LX4 also supports a range of 10 kilometres (6.2 mi) over SMF.

For MMF links the WDM output needs to be be coupled through a SMF offset-launch mode-conditioning patch cord. This is explained in subclauses 53.6 and 38.11.4 of the IEEE 802.3 spec.[10]

Until 2005 10GBASE-LX4 optical modules were cheaper than 10GBASE-LR optical modules.

10GBASE-LX4 was used by people who wanted to support both MMF and SMF with a single optical module.

10GBASE-LX4 is now an obsolete technology and has no significant market presence.

Copper

10G Ethernet can also run over twin-ax cabling, twisted pair cabling, and backplanes.

10GBASE-CX4

10GBASE-CX4 — was the first 10G copper standard published by 802.3 (as 802.3ak-2004). It uses the XAUI 4-lane PCS (Clause 48) and copper cabling similar to that used by InfiniBand technology. It is specified to work up to a distance of 15 m (49 ft). Each lane carries 3.125 G baud of signaling bandwidth.

10GBASE-CX4 offers the advantages of low power, low cost and low latency, but has a bigger form factor and more bulky cables than the newer single lane SFP+ standard and a much shorter reach than fibre or 10GBASE-T.

Shipments of 10GBASE-CX4 today are very low.[13]

SFP+ Direct Attach

Also known as 10GSFP+Cu, 10GBase-CR, or 10GBase-CX1, SFP+ Direct Attach uses a passive twin-ax cable assembly and connects directly into an SFP+ housing. It has a range of 7 m and like 10GBASE-CX4, is low power, low cost and low latency with the added advantages of using less bulky cables and of having the small form factor of SFP+. SFP+ Direct Attach today is tremendously popular, with more ports installed than 10GBASE-SR.[13]

Backplane

Backplane Ethernet — also known by its task force name 802.3ap — is used in backplane applications such as blade servers and routers/switches with upgradable line cards. 802.3ap implementations are required to operate in an environment comprising up to 1 metre (39 in) of copper printed circuit board with two connectors. The standard defines two port types for 10 Gbit/s (10GBASE-KX4 and 10GBASE-KR) and a 1 Gbit/s port type (1000BASE-KX). It also defines an optional layer for FEC, a backplane autonegotiation protocol and link training for 10GBASE-KR where the receiver can set a three tap transmit equalizer. The autonegotiation protocol selects between 1000BASE-KX, 10GBASE-KX4, 10GBASE-KR or 40GBASE-KR4 operation. 40GBASE-KR4 is defined in 802.3ba. [16]

New backplane designs use 10GBASE-KR rather than 10GBASE-KX4.[13]

10GBASE-KX4

This operates over four backplane lanes and uses the same physical layer coding (defined in IEEE 802.3 Clause 48) as 10GBASE-CX4.

10GBASE-KR

This operates over a single backplane lane and uses the same physical layer coding (defined in IEEE 802.3 Clause 49) as 10GBASE-LR/ER/SR.

10GBASE-T

10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft).[17] 10GBASE-T cable infrastructure can also be used for 1000BASE-T allowing a gradual upgrade from 1000BASE-T using autonegotiation to select which speed to use. 10GBASE-T has latency in the range 2 to 4 microseconds compared to 1 to 12 microseconds on 1000BASE-T.[18][19] As of 2010 10GBASE-T silicon is available from several manufacturers [20][21][22][23] with claimed power dissipation of 3-4 W at structure widths of 40 nm. and with 28 nm in development, power will continue to decline.[24]

Connectors

10GBASE-T uses the IEC 60603-7 8P8C (commonly known as RJ45) connectors already widely used with Ethernet. Transmission characteristics are now specified to 500 MHz.

Cables

Category 6A or better balanced twisted pair cables specified in ISO/IEC 11801 amendment 2 or ANSI/TIA-568-C.2 are needed to carry 10GBASE-T up to distances of 100 m. Category 6 cables can carry 10GBASE-T for shorter distances when qualified according to the guidelines in ISO TR 24750 or TIA-155-A.

Electrical characteristics

The 802.3an standard defines the wire-level modulation for 10GBASE-T as a Tomlinson-Harashima precoded (THP) version of pulse-amplitude modulation with 16 discrete levels (PAM-16), encoded in a two-dimensional checkerboard pattern known as DSQ128. Several proposals were considered for wire-level modulation, including PAM with 12 discrete levels (PAM-12), 10 levels (PAM-10), or 8 levels (PAM-8), both with and without Tomlinson-Harashima Precoding (THP). PAM-5 is what is used in the older 1000BASE-T gigabit Ethernet standard.

WAN PHY (10GBASE-W)

The WAN PHY uses the 10GBASE-S, 10GBASE-L and 10GBASE-E optical PMDs and is designated as 10GBASE-SW, 10GBASE-LW or 10GBASE-EW. Its Physical Coding Sublayer 64B/66B PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMDs in Clauses 52. It also uses a WAN Interface Sublayer (WIS) defined in Clause 50 which adds extra encapsulation to format the frame data to be compatible with SONET STS-192c.[10]

The WAN PHY was designed to interoperate with OC-192/STM-64 SDH/SONET equipment using a light-weight SDH/SONET frame running at 9.953 Gbit/s.

The WAN PHY is a niche product and rarely used.

10GbE NICs

10GbE network interface cards are available from several manufacturers. These plug into ordinary computer servers using PCI express and connect to the LAN with a choice of PHY modules.

See also

Notes and references

  1. ^ Anil Sharma (January 19, 2011). "LightCounting forecasts CAGR of Over 300 Percent for 10GBASE-T Port Shipments Through 2014". TMCnet. http://10-gigabit-ethernet.tmcnet.com/topics/10-gigabit-ethernet/articles/136124-lightcounting-forecasts-cagr-over-300-percent-10gbase-t.htm. Retrieved May 7, 2011. 
  2. ^ "Dell'Oro press release". http://www.delloro.com/news/2010/ES022210.htm/. 
  3. ^ "Intel blog about Interop 2011". http://communities.intel.com/community/openportit/server/blog/2011/05/12/overheard-at-interop-a-few-questions-about-ethernet. 
  4. ^ "LightCounting's LightTrends April 2010". http://www.lightcounting.com/lighttrends/1004/. 
  5. ^ "10GbE Optical Component and SFP+ Modules: This Time It's Different by Andrew Schmitt". http://www.nyquistcapital.com/2007/11/28/10gbe-and-sfp-this-time-its-different/. 
  6. ^ "The road to SFP+: Examining module and system architectures by Ryan Latchman and Bharat Tailor". http://lw.pennnet.com/display_article/317903/13/ARTCL/none/OTRAT/1/The-road-to-SFP+:-Examining-module-and-system-architectures/. 
  7. ^ "Optical Fiber and 10 gigabit Ethernet white paper by the 10GEA". http://www.10gea.org/optical-fiber-10ge.htm. 
  8. ^ "Why choose Multimode fiber? by Corning". http://www.corning.com/docs/opticalfiber/cn0603.pdf. 
  9. ^ "10 Gigabit Ethernet over Multimode Fiber by John George". http://bicsi.org/Events/Conferences/Spring/2005/GeorgePRES.pdf. 
  10. ^ a b c d e f g "IEEE 802.3 standard". http://standards.ieee.org/getieee802/802.3.html. 
  11. ^ "How to tell? MMF or SMF". http://www.velocityreviews.com/forums/t33419-how-to-tell-mmf-or-smf.html. 
  12. ^ "Description of Cisco 10G optical modules". http://www.cisco.com/en/US/prod/collateral/modules/ps5455/ps6574/product_data_sheet0900aecd801f92aa.html. 
  13. ^ a b c d e "Another Serving of Alphabet Soup - by Intel". http://communities.intel.com/community/openportit/server/blog/2011/06/20/10-gigabit-ethernet-update-another-serving-of-alphabet-soup. 
  14. ^ "10GBase-LX4 vs 10GBase-LRM: A debate". Archived from the original on 2009-07-20. http://lw.pennnet.com/display_article/249488/13/ARTCL/none/none/1/10GBase-LX4-vs-10GBase-LRM:-A-debate/. Retrieved 2009-07-16. 
  15. ^ Cisco 10GBASE XENPAK Modules
  16. ^ "IEEE P802.3ap Backplane Ethernet Task Force". http://grouper.ieee.org/groups/802/3/ap/index.html. 
  17. ^ "IEEE Standards Status Report for 802.3an". http://standards.ieee.org/cgi-bin/status?802.3an. 
  18. ^ 10GBASE-T for Broad 10 Gigabit Adoption in the Data Center, Intel, http://download.intel.com/support/network/sb/intel_ethernet_10gbaset.pdf, retrieved 2011-12-21 
  19. ^ SWITCHES SWITCH FROM 1000BASE‐T TO 10GBASE‐T NOW, Teranetics, October 2009, http://www.plxtech.com/files/pdf/support/10gbaset/whitepapers/10GBase-T_1000Base-T_Switches.pdf 
  20. ^ "Broadcom 10GBASE-T PHY". http://www.broadcom.com/products/Physical-Layer/10-Gigabit-Ethernet-PHYs. 
  21. ^ "PLX Technology, Teranetics 10GBASE-T PHY". http://www.plxtech.com/products/10gbase-t. 
  22. ^ "Solar Flare 10GBASE-T PHY". Archived from the original on 2009-09-07. http://www.solarflare.com/products/10xpress.php. Retrieved 2009-09-05. 
  23. ^ "Aquantia 10GBASE-T PHY". http://www.aquantia.com/pdf/AquantiaAQ1002_11172008.pdf. 
  24. ^ "Transceivers SFP+ 10GB BIDI". Fibesrstore. January 2005. http://www.fiberstore.com/transceivers-sfp-10gb-bidi-c-1_99.html. 

External links